Nonverbal multi-input and feedback devices for user intended computer control and communication of text, graphics and audio

ABSTRACT

There is disclosed devices, systems and methods for nonverbal multi-input and feedback devices for user intended computer control and communication of text, graphics and audio. The system comprises sensory devices comprising sensors to detect a user inputting gestures on sensor interfaces, a cloud system comprising a processor, for retrieving the inputted gestures detected by the sensor on the sensory device, comparing the inputted gestures to gestures stored in databases on the cloud system, identifying at least a text, graphics and/or speech command comprising a word that corresponds to the inputted gesture; showing the command to the user; and transmitting the command to another device.

RELATED APPLICATION INFORMATION

This application is a continuation of U.S. application Ser. No.17/561,541 titled NONVERBAL MULTI-INPUT AND FEEDBACK DEVICES FOR USERINTENDED COMPUTER CONTROL AND COMMUNICATION OF TEXT, GRAPHICS AND AUDIO,filed Dec. 23, 2021, which is a continuation U.S. application Ser. No.17/141,162 titled NONVERBAL MULTI-INPUT AND FEEDBACK DEVICES FOR USERINTENDED COMPUTER CONTROL AND COMMUNICATION OF TEXT, GRAPHICS AND AUDIO,filed Jan. 4, 2021, now U.S. Pat. No. 11,237,635, which is acontinuation-in-part of the following applications:

U.S. patent application Ser. No. 15/498,158, filed Apr. 26, 2017,entitled “Gesture Recognition Communication System”;

U.S. patent application Ser. No. 16/749,892, filed Jan. 22, 2020,entitled “CONTEXT AWARE DUAL DISPLAY TO AUGMENT REALITY,” which claimspriority from Provisional application No. 62/704,048, filed on Jan. 22,2019;

U.S. patent application Ser. No. 15/929,085, filed Jan. 9, 2019,entitled “BRAIN COMPUTER INTERFACE FOR AUGMENTED REALITY” which claimspriority from Provisional application No. 62/752,133, filed on Oct. 29,2018;

the contents all of which are incorporated herein by reference.

NOTICE OF COPYRIGHTS AND TRADE DRESS

A portion of the disclosure of this patent document contains materialwhich is subject to copyright protection. This patent document may showand/or describe matter which is or may become trade dress of the owner.The copyright and trade dress owner has no objection to the facsimilereproduction by anyone of the patent disclosure as it appears in thePatent and Trademark Office patent files or records, but otherwisereserves all copyright and trade dress rights whatsoever.

BACKGROUND Field

This disclosure relates to nonverbal multi-input and feedback devicesfor user intended computer control and communication of text, graphicsand audio.

Description of the Related Art

Hundreds of millions of people around the world use body language tocommunicate, and billions of people have difficulty interpreting theirneeds.

Advancements in technology have allowed individuals with speechdisabilities to use technical devices to communicate. Smart devicesallow individuals ease of interacting with devices by simply touching ascreen using a finger, stylus, or similar apparatus.

However, while technology has advanced to allow ease of interactionusing touchscreens, individuals with speech disabilities still facechallenges communicating with others using spoken words. Therefore,there is a need for a unified system to allow an individual tocommunicate with others through spoken word by interacting with acomputing device via personalized access methods that are allinteroperable with the unified system.

DESCRIPTION OF THE DRAWINGS

The patent or application file contains at least one drawing executed incolor. Copies of this patent or patent application publication withcolor drawing(s) will be provided by the Office upon request and paymentof the necessary fee.

FIG. 1A is a block diagram of a gesture recognition communicationsystem.

FIG. 1B is a block diagram of nonverbal multi-input and feedback devicesfor user intended computer control and communication.

FIG. 1C is a block diagram of nonverbal multi-input and feedback devicesand input connections for user intended computer control andcommunication.

FIG. 2 is a block diagram of a computing device.

FIG. 3 is a block diagram of a sensory device.

FIG. 4 is a flowchart of using the gesture recognition communicationsystem to generate speech commands and/or of using nonverbal multi-inputand feedback devices for user intended computer control andcommunication.

FIG. 5 is a flowchart for configuring a new gesture to be used in thegesture recognition communication system and/or nonverbal multi-inputand feedback devices.

FIG. 6 is a sample of pre-configured gestures that may exist in thesystem and/or nonverbal multi-input and feedback devices.

FIG. 7A is a display of a sensory device for a user to input a gesture.

FIG. 7B is a display of a nonverbal multi-input and feedback device fora user to input a gesture.

FIG. 8A is a display of a sensory device for a user to access thepre-configured gestures.

FIG. 8B is a display of a nonverbal multi-input and feedback device fora user to access the pre-configured gestures.

FIG. 9 is a display of a sensory device or a nonverbal multi-input andfeedback device for a user to customize a gesture.

FIG. 10 is a block diagram of a nonverbal multi-input and feedbackdevice.

FIG. 11 is a block diagram of a single framework for translating diversesensor inputs into a variety of understandable communication and commandoutputs.

FIG. 12 is a table showing example input/output (I/O) and associatedinformation for a nonverbal multi-input and feedback device.

FIG. 13 is a diagram showing flow diagram and related input outputexample of tap and/or swipe tracking for a nonverbal multi-input andfeedback device.

FIG. 14 is a diagram showing flow diagram and related input outputexample of breathe tracking for a nonverbal multi-input and feedbackdevice.

FIG. 15 is a diagram showing flow diagram and related input outputexample of face and eye tracking for a nonverbal multi-input andfeedback device.

FIG. 16 is a diagram showing flow diagram and related input outputexample of head and neck tracking for a nonverbal multi-input andfeedback device.

FIG. 17 is a diagram showing flow diagram and related input outputexample of brain and biosignal tracking for a nonverbal multi-input andfeedback device.

FIG. 18 is a diagram showing flow diagram and related input outputexample of multimodal combination of sensors for a nonverbal multi-inputand feedback device.

FIG. 19 is a flow diagram showing a closed loop biosignal data flow fora nonverbal multi-input and feedback device.

FIG. 20 is a flow diagram showing multimodal, multi-sensory system forcommunication and control for a nonverbal multi-input and feedbackdevice.

FIG. 21 is a block diagram showing an example of cloud processing for anonverbal multi-input and feedback device.

FIG. 22 is a block diagram showing an example of a system architecturefor integrated virtual AI assistant and web services for a nonverbalmulti-input and feedback device.

FIG. 23 is a block diagram showing an example of system operations for anonverbal multi-input and feedback device

Throughout this description, elements appearing in figures are assignedthree-digit reference designators, where the most significant digit isthe figure number and the two least significant digits are specific tothe element. An element that is not described in conjunction with afigure may be presumed to have the same characteristics and function asa previously-described element having a reference designator with thesame least significant digits.

DETAILED DESCRIPTION

Described herein is a gesture recognition communication system and/ornonverbal multi-input and feedback device used to enhance a human'scapacity to communicate with people, things and data around themremotely over a network or virtually within similar environments. Thissystem or device will benefit individuals with communicationdisabilities. In particular, it will benefit nonverbal individuals,allowing them to express their thoughts in the form of spoken languageto allow for easier communication with other individuals, providing avariety of sensory input modes that can be adapted to various physicalor cognitive disabilities, where individuals can communicate with theirhands, eyes, breathe, movement and direct thought patterns. Gesture, asused throughout this patent, may be defined as a ‘time-based’ analoginput to a digital interface, and may include, but not be limited to,time-domain (TD) biometric data from a sensor, motion tracking data froma sensor or camera, direct selection data from a touch sensor,orientation data from a location sensor, and may include the combinationof time-based data from multiple sensors.

Description of Apparatus

Referring now to FIG. 1A, there is shown a block diagram of anenvironment 100 of a gesture recognition communication system. Theenvironment 100 includes sensory devices 110, 120, 130, 150, and 160,and a cloud system 140. Each of these elements are interconnected via anetwork (not shown).

The sensory devices 110, 120, 130, 150 and 160, are computing devices(see FIG. 3 ) that are used by users to translate a user's gesture to anaudible, speech command. The sensory devices 110, 120, 130, 150 and 160sense and receive gesture inputs by the respective user on a sensorinterface, such as a touchscreen, or peripheral sensory device used asan accessory to wirelessly control the device. The sensory devices 110,120, 130, 150 and 160 also generate an audio or visual output whichtranslates the gesture into a communication command. The sensory devices110, 120, 130, 150 and 160, may be a tablet device, or a smartwatch, ora similar device including a touchscreen, a microphone and a speaker.The touchscreen, microphone and speaker may be independent of orintegral to the sensory devices 110, 120, 130. Alternatively, thesensory devices may be screenless devices that do not have a speaker,but contain one or more sensors, such as smart glasses as seen insensory device 150, or a brain computer interface, as seen in sensorydevice 160. For purposes of this patent, the term “gesture” means auser's input on a touchscreen of a computing device, using the user'sfinger, a stylus, or other apparatus including but not limited towirelessly connected wearable or implantable devices such as a BrainComputer Interface (BCI), FMRI, EEG or implantable brain chips, motionremote gesture sensing controllers, breathing tube sip and puffcontrollers, electrooculography (EOG) or eye gaze sensing controllers,to trigger a function.

FIG. 1B is a block diagram of nonverbal multi-input and feedback devicesand inputs for user intended computer control and communication. FIG. 1Badds example sensor inputs to the sensor devices of FIG. 1A, howeverthere may be additional inputs to the sensors. The input examples shownare analog biodata face input 105 to devices 110 and 120; analog biodatatouch input 115 to devices 110, 120 and 130; analog biodata brain waveinput 125 to device 150; analog biodata brain wave muscle input 135 todevice 150; analog biodata eye input 145 to device 160; and analogbiodata head movement input 155 to device 160. Input 125 may includeinputs to a device 150 that is an implanted sensor. Input 135 mayinclude inputs to a device 150 that is used anywhere the user has amuscle like a leg, foot, tongue, cheek, rectum.

FIG. 1C is a block diagram of nonverbal multi-input and feedback devicesand input connections for user intended computer control andcommunication. FIG. 1C adds example sensor input connections to theinputs and sensor devices of FIG. 1A-B, however there may be additionalinput connections. The input connection examples shown include aconnection between sensory devices 120 and 150 so data received by oneof these sensors can be in addition or in the alternative be receive bythe other sensor. For example, biodata 125 may be received by both oreither of sensors 120 and 150. This is also true for input 105, 115 and135. The input connection examples shown also include a connectionbetween sensory devices 120 and 160 so data received by one of thesesensors can be in addition or in the alternative be receive by the othersensor. For example, biodata 145 may be received by both or either ofsensors 120 and 160. This is also true for input 105, 115 and 155.

The cloud system 140 is a computing device (see FIG. 2 ) that is used toanalyze a user's raw input into a sensory device to determine a speechcommand to execute. The cloud system 140 develops libraries anddatabases to store a user's gestures and speech commands. The cloudsystem 140 also includes processes for 1D, 2D, 3D, and 4D gesturerecognition algorithms. The cloud system 140 may be made up of more thanone physical or logical computing device in one or more locations. Thecloud system 140 may include software that analyzes user, network,system data and may adapt itself to newly discovered patterns of use andconfiguration.

Turning now to FIG. 2 there is shown a block diagram of a computingdevice 200, which can be representative of any of the sensory devices110, 120, 130, 150 and 160, and the cloud system 140 in FIG. 1 . Thecomputing device 200 may be any device with a processor, memory and astorage device that may execute instructions including, but not limitedto, a desktop or laptop computer, a server computer, a tablet, asmartphone or other mobile device, wearable computing device orimplantable computing device. The computing device 200 may includesoftware and/or hardware for providing functionality and featuresdescribed herein. The computing device 200 may therefore include one ormore of: logic arrays, memories, analog circuits, digital circuits,software, firmware and processors. The hardware and firmware componentsof the computing device 200 may include various specialized units,circuits, software and interfaces for providing the functionality andfeatures described herein. The computing device 200 may run an operatingsystem, including, for example, variations of the Linux, MicrosoftWindows and Apple Mac operating systems.

The computing device 200 has a processor 210 coupled to a memory 212,storage 214, a network interface 216 and an I/O interface 218. Theprocessor 210 may be or include one or more microprocessors, fieldprogrammable gate arrays (FPGAs), application specific integratedcircuits (ASICs), programmable logic devices (PLDs) and programmablelogic arrays (PLAs). The computing device 200 may optionally have abattery 220 for powering the device for a usable period of time andcharging circuit 222 for charging the battery.

The memory 212 may be or include RAM, ROM, DRAM, SRAM and MRAM, and mayinclude firmware, such as static data or fixed instructions, BIOS,system functions, configuration data, and other routines used during theoperation of the computing device 200 and processor 210. The memory 212also provides a storage area for data and instructions associated withapplications and data handled by the processor 210.

The storage 214 provides non-volatile, bulk or long term storage of dataor instructions in the computing device 200. The storage 214 may takethe form of a magnetic or solid state disk, tape, CD, DVD, or otherreasonably high capacity addressable or serial storage medium. Multiplestorage devices may be provided or available to the computing device200. Some of these storage devices may be external to the computingdevice 200, such as network storage or cloud-based storage. As usedherein, the term storage medium corresponds to the storage 214 and doesnot include transitory media such as signals or waveforms. In somecases, such as those involving solid state memory devices, the memory212 and storage 214 may be a single device.

The network interface 216 includes an interface to a network such as thenetwork described in FIG. 1 . The network interface 216 may be wired orwireless.

The I/O interface 218 interfaces the processor 210 to peripherals (notshown) such as a graphical display, touchscreen, audio speakers, videocameras, microphones, keyboards and USB devices.

Turning now to FIG. 3 there is shown a block diagram of a sensory device300, which can be representative of any of the sensory devices 110, 120,130, 150 and 160, of FIG. 1 . The processor 310, memory 312, storage314, network interface 316 and I/O interface 318 of FIG. 3 serve thesame function as the corresponding elements discussed with reference toFIG. 2 above. These will not be discussed further here. The computingdevice 300 may optionally have a battery 328 for powering the device fora usable period of time and charging circuit 326 for charging thebattery. Charging circuit 326 may be used for charging the batteries ofmultiple sensory devices together. For example, a BCI/AR device, canhave a single charging circuit used to charge both the BCI and the ARsystem which have their own batteries, but since they're two sensorydevices combined into a single wearable, charging can be simplified forthe user with a single charging circuit.

The sensor 320 can include any sensor designed to capture data. Thesensor 320 can be a touch sensor, a camera vision sensor, a proximitysensor, a location sensor, a rotation sensor, a temperature sensor, agyroscope, an accelerometer. The sensor 320 can also include abiological sensor, an environmental sensor, a brainwave sensor, or anacoustic sensor. The sensory device 300 can include a single sensor ormultiple sensors with a combination of various types of sensors.

The speaker 322 can be a wired or wireless speaker integrated into thesensory device 300, or attached to, or wirelessly connected to, thesensory device 300. The speaker 322 allows the sensory device to outputthe translated gesture into a speech command.

The actuator 324 may provide user feedback to the system. For example,the actuator may be used for physical actuation in the system, such ashaptic, sound or lights.

Description of Processes

Referring now to FIG. 4 , there is shown a process 400 of using thegesture (or sensory) recognition communication system, such as thesystem shown in FIG. 1A, to generate speech commands; and/or of usingnonverbal multi-input and feedback devices, such as the system shown inFIG. 1B or 1C, for user intended computer control and communication oftext, graphics and audio. The process occurs on the sensory device 410,as well as the cloud system 440. While the process includes steps thatoccur on both the sensory device 410 and the cloud system 440, theprocess can also be performed locally on just the sensory device 410. Inthis case, the actions performed by cloud system 440 are performed bythe sensory device. Sensory device 410 represents any of sensory devices110-160. Process 400 should be able to execute; entirely on thesensory/input device (not shown); partially on the sensory/input deviceand partially in cloud (shown).

The process 400 begins with at 415 with a user activating the gesture orsensory recognition communication system. The activation can occur whena user logs into his account on an app stored on the sensory device 410.After the user has logged into his account, the process proceeds to 420where the user inputs a gesture. Alternatively, a user can begin usingthe system without logging into an account.

The gesture can include a single tap on the touchscreen of the sensorydevice 410. Alternatively, the gesture can include a swipe in a certaindirection, such as swipe up, swipe down, swipe southeast, and such. Inaddition, the gesture can include a letter, or a shape, or an arbitrarydesign. The user can also input a series of gestures. The sensors on thesensory device 410 capture the gestures inputted and executes all theprocesses locally on the sensory device or transmits the raw data of thegesture inputted to the cloud system. The inputted gesture may be storedin the storage medium on the sensory device 410, and synchronized to thecloud system 440.

After the user inputs his gesture, the process proceeds to 425, wherethe gesture is transmitted over a network to the cloud system. The cloudsystem retrieves the inputted gesture at 425, and then compares theinputted gesture to a gesture database either locally or on the cloudsystem that stores preconfigured gestures. The cloud system may analyzethe raw data of the gesture inputted by determining the pattern, such asthe direction of the gesture, or by determining the time spent in onelocation, such as how long the user pressed down on the sensory device.For example, if the user inputs a swipe up gesture, then the raw datawould indicate a continuous movement on the sensor interface of thesensory device. Alternatively, if the user inputted a double tap on thesensor interface, then the raw data would indicate a similar positionwas pressed for a short period of time. The cloud system would analyzethe raw data to interpret the inputted gesture. After the raw data hasbeen interpreted, the cloud system would compare the raw data inputtedto a database or library of previously saved gestures stored on thecloud system. The database or library would include previously savedgestures with corresponding communication commands associated with eachpreviously saved gesture. The database or library may be specific to acertain user, thereby allowing one user to customize the gestures tomean particular communication commands of his choice, while another usercan use the preconfigured gestures to translate into differentcommunication commands. For example, one user may desire to customizethe swipe up gesture to mean, “Yes”, while another user may customizethe swipe up gesture to mean, “No.” Therefore, every user may have aunique gesture database associated with his user account. As noted, insome cases processes 425-470 are performed by the sensor device.

The cloud system 440 determines if there is a gesture match at 435between the inputted gesture and the stored preconfigured gestures. Todetermine if there is a gesture match, the cloud system would analyzethe inputted gesture, and the raw data associated with the inputtedgesture, and lookup the preconfigured gestures stored in the database.If the inputted gesture exists in the database, then the database willretrieve that record stored in the database. The record in the databasewill include the communication command associated with the inputtedgesture. Alternatively, if no communication is associated with a savedgesture, the system may transmit a null or empty message, as seen in450, which may include data associated with the transmission includingbut not limited to raw user input data which may be saved in thedatabase.

If the cloud system does not locate a match, meaning the cloud systemdid not locate a record in the database of preconfigured gestureslooking like the inputted gesture, then the process 400 proceeds to 445where the unidentified gesture is stored in the cloud system 440. Thecloud system 440 stores the unidentified gesture in a database to allowthe cloud system to improve on the gesture pattern recognition overtime. As a user interacts with the gesture or sensory recognitioncommunication system, the system will develop pattern recognitionlibraries that are based on the user's inputted gestures. For example,one user may press his finger on the sensor interface for 2 seconds toindicate a “long hold” gesture, while another user may press his fingeron the sensor interface for 3 seconds to indicate a “long hold”. Thedatabase may be configured to identify a “long hold” gesture afterpressing on the sensor interface for 4 seconds. In this case, both ofthe users' “long hold” gesture may not be found in the gesture database,because the database was configured with different requirements for the“long hold” gesture. Therefore, over time, as a user continues to pressthe sensor interface for 2 seconds, the database will update itself andrecognize that the user is attempting to input the “long hold” gesture.

After the unidentified gesture is stored, the cloud system transmits anempty message at 450 to the sensory device 410. The sensory device 410then displays an “empty” message at 460. The “empty” message may be aspeech command that says, “The system does not understand that gesture.”Alternatively, the message might be an emoji showing that the system didnot understand the gesture, or simply delivers an undefined message “_”.After 460, the unidentified gesture is saved as a new gesture at 480based on the system identifying an unmapped gesture+message. A user maymap a new ‘saved gesture’ to a desired communication or control commandmanually, or the system may map the new gesture to a new communicationor command automatically based on machine learning, for future use.Refer to FIG. 5 for the “save new gesture” flow diagram.

Alternatively, if the cloud system did locate a match between theinputted gesture and the stored preconfigured gestures, then the process400 proceeds to 465 to retrieve the communication command. Thecommunication command is retrieved and identified when the databaseretrieves the stored record in the database of the gesture. For eachgesture stored in the database, there will be a communication commandassociated with the gesture. The communication command can be a naturallanguage response, such as “Yes” or “No”. Alternatively, thecommunication command can be a graphical image of an object, such as anemoji of a happy face, or other actuation including but not limited to aphotograph, a color, animated picture or light pattern, a sound, or avibration pattern. After the communication command has been identified,the cloud system 440 then transmits the communication command at 470over the network to the sensory device 410. The sensory device 410 thengenerates the speech command 475. In addition, the sensory device maydisplay a graphical image, or other actuation described above, if thatwas what the inputted gesture was to be translated to. If thecommunication command is a word or phrase, then the sensory device willgenerate a speech command, in which the speaker on the sensory devicewill generate the speech saying the words or phrase associated with thegesture. The communication command may also contain contextual data thatis appended to or modifies the communication being transmitted.Contextual data may include contact lists, location, time, urgencymetadata.

Referring now to FIG. 5 , there is shown a process 500 for configuring anew gesture to be used in the gesture (or sensory) recognitioncommunication system, such as the system shown in FIG. 1A; and/ornonverbal multi-input and feedback devices, such as shown in FIG. 1B or1C. Process 500 can be performed on any of the sensor systems 110-160.Notably, the new gesture and configuring can be responsive to gesturesinput into one of the sensor systems.

The process 500 begins when a user initiates the new gesture creationprocess. This can occur when the user selects a new gesture icon thatexists on the sensor interface of the sensory device. After the processhas been initiated, the user can input a new gesture at 515. The newgesture can be any gesture that is not included in the pre-configuredgestures. At 520, the system determines if the new gesture has beencompletely inputted.

If the gesture has not been completed inputted, then the process returnsto 515 to allow the user to complete inputting the new gesture.Alternatively, the user can enter a series of gestures.

If the gesture has been completed inputted, then the process proceeds to525 where the system asks the user if the user wants to preview the newgesture. If the user does want to preview it, then the new gesture isdisplayed at 530 for the user to preview. If the user does not want topreview the new gesture, then the system asks the user if the user wantsto save the new gesture at 535. If the sensory device 510, is connectedto the cloud system, then at 560, it sends the recorded gesture to thecloud system to be analyzed and categorized.

If the user wants to save the new gesture, then the new gesture is savedat 540 in the gesture database stored on the cloud system. The systemnext determines at 545 if the user wants to configure the new gesturewith a communication command. If the user does not want to configure thenew gesture at that moment, then the process ends. The user can chooseto configure the new gesture at a later time. Alternatively, if the userwants to configure the new gesture, then the process proceeds to 550,where the user adds a communication command to the new gesture. Thecommunication command can be words or phrases in a natural language.Alternatively, the communication command can be a graphical image, orother actuation pattern (such as light, color, sound, vibration). Afterthe communication command has been stored in the gesture database, theprocess ends.

Referring to FIG. 6 , there is shown a sample 600 of pre-configuredgestures that may exist in the gesture recognition communication system,such as the system shown in FIG. 1A; and/or nonverbal multi-input andfeedback devices, such as shown in FIG. 1B or 1C. The pre-configuredgestures may include a single tap (see red dot), a double tap, a longhold. In addition, the pre-configured gestures may include swipe up,swipe down, swipe left, swipe right, swipe northeast, swipe northwest,swipe southeast, swipe southwest. The pre-configured gestures can alsoinclude combinations of taps and swipes, such as the up and hold gestureshown, and can also include letters, numbers, shapes, and anycombination of those. The pre-configured gestures shown are someexamples for just the touch I/O sensory devices. The pre-configuredgestures can also include data on thought, breaths, glances, motiongestures, and similar nonverbal gestures. These physical gestures arenot limited to touch or swipe, but could also be used with head/neckmovement, face/eye tracking movement, etc. Refer above to a definitionof “gesture” as a time based input with a beginning/middle/end.Regarding I/O that are biosignals (EEG, EMG, etc), then the geometricshape of a gesture may not apply. Instead, a gesture may be or includeprocessed time-based frequency neuronal data that is associated withobjects to interact with them.

Referring to FIG. 7A, there is shown a display of a sensory device 710,such as sensory device 110 in FIG. 1A. The sensory device may be used bya user to input a gesture. The sensory device 710 may displayinformation about the user at 720. In addition, the sensory deviceincludes a sensor interface 730 for the user to input a gesture. Thesensory device 710 also includes translated text at 740. The translatedtext may display the natural language, or other information attributesassociated with the gesture inputted into the sensor interface. Ifreceiving a message from a connected contact across a network, then thesender's message is displayed and spoken aloud as it was configured fromthe sender, which may also include data about the sender. The sensorydevice 710 also includes the speech command 750. The speech command 750is the spoken natural language for the gesture that was inputted by auser. The sensory device may also provide user feedback to the system,including physical actuation elements, such as haptic, lights, orsounds.

Referring to FIG. 7B, there is shown a nonverbal multi-input andfeedback device 702, such as shown in FIG. 1B or 1C. FIG. 7B addsexample features of device 702 to those of device 710, however there maybe fewer or more features than those show in device 702 that are addedto device 710. The feature examples include optionally battery status715 for showing the battery power for powering the device for a usableperiod of time. They also include saved items or phrases 745, such asfor saving new gestures 480 input to the sensors to be used to correctlyinterpreted the nonverbal multi-input for user intended computer controland communication. Next, they include I/O settings 755 to be used by thedevice to configure the sensors and device to correctly interpreted thenonverbal multi-input for user intended computer control andcommunication. They also include output method options 765 to selectoutput options be used by the device to output the user intendedcomputer controls and communication of text, graphics and audio to othercomputing devices. Options 765 may include multiple buttons/methods foroutput selection (e.g., say it, send it, save it).

The feature examples also include biofeedback 735 for feeding back tothe user information (e.g., nonverbal multi-inputs), text, graphicsand/or audio so that the user can correctly input data to the sensorsthat can be interpreted as the user intent for user intended computercontrol and communication. Biofeedback can be an important and a part ofthe interface. It may include feedback to the user in response to userinputs to the sensors. The feedback may be visual, auditory and/orhaptic. For example, visual may be or include a configurable cursor, ora comet trail after swiping, or a progress bar around an object fordwell time or concentration, or stimulation of the user's visual brain.Auditory feedback may be or include sound effects based on directinteraction with sensory device, time-based progress of interaction suchas a rising tone as during concentration or dwell, or system prompts toalert user to take actions, or stimulation of the user's visual brain.Haptic feedback may be or include vibration patterns based on directinteraction with the sensory device, time-based progresss of interactionsuch as different vibration patterns for correct versus incorrectinteraction, or system prompts to alert user to take actions, orstimulation of the user's visual brain). Spatial audio and hapticsbiofeedback may include auditory feedback like sound effects based ondirect interaction with sensory device, time-based progress ofinteraction such as a rising tone as during concentration or dwell, orsystem prompts to alert user to take actions, stimulation of the user'svisual brain, or sounds presented to the user that are spatially placedand can be identified by the user based on how their brain perceives itsposition (e.g. left, right, close far, front, back, moving from left toright then up). Haptic biofeedback may be or include vibration patternsbased on direct interaction with the sensory device, time-based progressof interaction such as different vibration patterns for correct versusincorrect interaction, or system prompts to alert user to take actions,stimulation of the user's visual brain, or haptic vibration patternspresented to the user that are felt or perceived by the brain spatiallyaround their body (e.g. left, right, close, far, strong, weak, front,back, moving from left to right then up).”

The feature examples also include network status 725 for showing thestatus of a network the device is communicating over such as wireless,wired, WIFI, internet, cell or another network, to send the userintended computer controls and/or communication to other computingdevices. The device has the proper equipment for communicating controlsignals and/or the communication over the network.

Referring to FIG. 8A, there is shown a display of a sensory device 810for a user to access the pre-configured gestures in the system ordevice, such as those of FIGS. 1A-C. The sensory device 810 shows auser's pre-configured gestures that are stored in a user's account. Thesensory device displays information about the user at 820. The sensorydevice 810 also displays the settings 830 that are configured for theuser 820. The settings 830 include settings such as taps 840, swipes855, diagonals 865, additional gestures 875, thought gestures 882,eyeglance gestures 886, motion gestures 890, breath gestures 894, andcreate new gesture 898. The taps 840 can include a single tap 845, adouble tap 850, or long hold, or any other taps. Each of the taps maytranslate into different words, phrases or sentences. For example, asingle tap 845, may translate into the words, “Thinking of You.” Adouble tap may translate into the words, “How are you?”

The swipes 855 may include swipe up, swipe down, swipe to the right,swipe to the left. Each of these swipes may translate into differentwords or phrases. For example, swipe up shown at 860 may mean “Yes”,while swipe down might mean “No.” Swipe gestures may include multi-touchand time elapsed such as “swipe and hold.”

The pre-configured gestures may also include diagonals shown at 865. Forexample, swipe northeast shown at 870 may mean, “Swipe northeast.” Inaddition, the pre-configured gestures may also include additionalgestures shown at 875. For example, shapes, letters, numbers and similarobjects may all be included in the pre-configured gestures. A gesture ofa rectangle shown at 880 may translate to “Rectangle.”

The thought gestures 882 may include various thoughts of a user. Forexample, a user's thoughts might include the thought of “Push”, shown at884, or “straight”. If the user thinks of the word “Push”, then thesystem may speak the word, “Push.”

The eye glance gestures 886 may include various eye movements of a user.For example, a user may “blink once”, as shown in 888, and that maycause the system to speak the word, “Yes.”

The motion gestures 890 may include movements made by a user. Forexample, a user may shake his head, as shown in 894, and the system maythen speak the word, “No.”

The breath gestures 894 may include information about a user's breathingpattern. For example, a user may breathe in a “puff” manner, and thesystem would detect that and may speak the word, “Help.”

A user can also create a new gesture at 898. For example, a user mayhave a touch based pattern, or a thought pattern that has not beenpreviously saved in the system. A user can customize new gestures withthe create new gesture option shown at 898.

A user 820 can refer to the setting 830 to determine how each gesturewill be translated. If a user added new gestures, as described by theprocess shown in FIG. 5 , then the new gesture and it's translatedlanguage will also appear in the list of gestures shown in the settings.

FIG. 8A may show a single display device 810 example to be used by auser to configure a variety of inputs for a variety of sensory devices,such as those of FIGS. 1A-C. Device 810 may provide or represent a1-to-1 mapping between a single user input and a single output. Alsocontemplated is an interface to configure a sensory device and/ornonverbal multi-input and feedback device to generally control an entireapplication like a mouse or keyboard.

FIG. 8B is a display 802 of a nonverbal multi-input and feedback devicefor a user to access the pre-configured gestures. Display 802 may bethat of a sensory device to personalize various aspects of the sensoryI/O of the sensory device to make it more natural to use with thesensory communication system. This example is multi-modal in that itallows settings for both touch interaction plus facial and eye trackingin a single personalization interface or nonverbal multi-input andfeedback device. Display 802 shows menus such as a settings menu 803 forselecting to change a keyboard layout 804, to customize trackingsettings 805 and to access controls 806. The customize tracking settings805 menu leads to pointer selection 807, selection method 808 andalternative tracking options (gaze type) 809. There is menu 803 also hascheck calibration menu 810 leading to calibration menus 811-813 forenabling a user to calibrate their abilities to a sensory device, orcalibrate a sensory device to receive gaze type eye inputs (see red eyeshapes on screen). The menu 803 also has an advanced setting menu 815.Menu 815 may include advanced I/O settings having the ability tomanually, or automatically via the system's machine learning subsystems,adjust a plurality of variables related to the biofeedback and userinterface responsiveness to the user's input and output experience. Inthe case of face, eye, head and neck movement and neural sensorydevices, these variables may include adjusting Pointer size, Pointerspeed, Pointer selection area, Hover delay timing, Button selection timewindow, tracking velocity and smoothing, vibration offsets, Responsedelay, Blink/Smile or other facial or biometric selection delay, Shortblink timer to reduce false positive selections which may apply to otherbiosignal sensory device inputs. The advanced settings may be availableinitially with default settings, and individually be adjusted to perfectthe user's preferences. These settings may be saved locally on thesensory device and/or synchronized to the cloud for future use,retrieval on other devices, or contributing data sets for machinelearning and new optimized models and software creations that may bedownloaded to the respective sensory device. These settings can be resetindividually or collectively.

Referring to FIG. 9 , there is shown a display of a sensory device 910for a user 920 to customize a gesture, such as sensory and/or anonverbal multi-input and feedback device of FIGS. 1A-C. The system ordevice comes pre-configured with gestures and translated phrases. A usercan choose to add new gestures to the system, or modify the phrase thatcorresponds to the pre-configured gestures. For example, a user may wishto change the meaning of the swipe up gesture to mean, “Happy.” Tomodify the phrase, the user will select the swipe up gesture shown in940. Where it says, “Yes”, the user 920 can delete that, and insert,“Happy.” The system then updates the gesture database such that wheneverthe user 920 swipes up, the system says, “Happy.” In addition, the user920 may modify the actuations and attributes associated with thegesture. For example, the user can modify the color 960, the vibrations970, the sounds 980, or the image 990 associated with the gesture.Alternatively, the user 920 can modify the swipe up gesture to displayan image of a happy face, or any visual image, or emoji. If emoji orvisual image contains descriptive text, that image will be spoken. Forexample, a visual image of a car will also include the spoken word “car”when displayed. The user 920, can also modify the language used by thesystem. If the user is a French speaker and wants to communicate inFrench, then the user 920 can update the language 950 to French, insteadof English which is shown. When the language is updated, then thepre-configured gestures will translate the gestures to words and phrasesin French.

FIG. 10 is a block diagram 1010 of a nonverbal multi-input and feedbackdevice such as herein. It may be a block diagram of a portion of thedevice such as a processing portion of the device. FIG. 10 may be ahigh-level system architecture block diagram of any of FIGS. 1A-C, thathelps explain that the major building blocks. Diagram 1010 can beapplied to the overall system (e.g. multiple devices used as inputs,into a common universal application interface that enables ourapplication (center) to synchronize data coming from multipledevices—see FIG. 11 —and process signals with meta data, plus vocabularyand output logic to a plurality of output methods. FIG. 11 takes this toa finer level of detail.

In the center of diagram 1010 is the application or main processingblock. To the left is the multimodal input and intent detection blockwhich receives and processes user inputs from sensors (e.g., based onuser input received by the sensors) such as touch; biosignals; keyboard;facial tracking; eye and pupil tracking; and alternative inputs. Thisblock feeds the processing from these inputs to the application. Aboveis a context and awareness block which receives and processes metadatainputs from sensors such as biometrics; environment; object recognition;facial recognition; voice recognition date and time; history; location;proximity and other metadata inputs. This block feeds the processingfrom these inputs to the application. To the right is an output andaction block which sends outputs to displays, computing devices,controllers, speakers and network communication devices such as flatscreen display; augmented/virtual reality; virtual AI assistant;synthesized voice; prosthetic device; social media and messaging; mediaconsumption and other outputs. The outputs may include control commandsand communication sent to other computing devices. they may includetext, graphics, emoji, and/or audio. Below is a vocabulary block thatprovides a lexicon or vocabulary in the selected language to theapplication. FIG. 10 may also be applied to a single sensory device untoitself. This may be a “BIG IDEA” in so far as the architecture can scalefrom a single closed-loop system (such as in FIGS. 13-17 , plus 19) aswell as combinations of sensory I/O devices (FIGS. 12, 18, 20 ). It maybe a system of systems that scale up, down and play together.

The system in diagram 1010 only requires 1 (or more) sensory input, 1intent detection api, 1 application, 1 (or more) meta data, 1 (or more)vocabulary, 1 (or more) output and action method, and 1 (or more)output/actuation system or device. It may be thought of as a universal“augmented intelligence” engine that takes inputs, enriches them withextra meaning, and directs the output based on instructions for theenriched information. The storyboard in FIG. 16 illustrates the power ofthis.

In a simple embodiment of diagram 1010, a user sees a symbol or buttonthat means “help”, and presses it, and the device says “help”. In a morecomplicated embodiment of diagram 1010, a user sees a symbol or buttonthat means “help”, and press it. Here, rather than the device saying“help”, it learns that the user is connected to a caregiver with logicto send urgent matters to that person via text or instant message whenaway from home. The device may geolocation data that indicates the useris away from home; tag the communication with appended contextualinformation; and its output and action logic tells the system to send atext message to the caregiver with the user's location in ahuman-understandable grammatically correct phrase “Help, I'm in OakPark” including the user's Sender ID/Profile and coordinates pinned on amap.

FIG. 11 is a block diagram 1110 of a single framework of a nonverbalmulti-input and feedback device such as herein. The block diagram 1110may be of a single framework for translating diverse sensor inputs intoa variety of understandable communication and command outputs for anonverbal multi-input and feedback device such as herein. FIG. 11 maydescribe in more detail what kind of processing is happening within andacross the blocks of FIG. 10 . Specifically, the left intention signalsbeing combined with context awareness metadata to enrich the data inorder to determine the logic of the output and action. FIG. 11 mayinclude the description of the Vocabulary and application boxes of FIG.10 , though not shown. It may be a block diagram of a portion of thedevice such as a processing portion of the device. In the framework,input from the sensors (e.g., due to input received by the sensors) arereceived by or as an input gesture. In the framework, context awarenessis used to interpret or determine the user gesture or intent from theinputs received. In the framework machine learning is used to interpretor determine the user gesture or intent from the inputs received. In theframework, output expression is used to determine the outputs, such ascontrol commands and communication sent to other computing devices thatinclude text, graphics, emoji, and/or audio. In the framework,destination is used to determine where the outputs are sent, such as towhat other computing devices the command and/or communications are to besent (such as by the network). The user's Primary and Secondary languagepreferences are accessed during the processing of intention data whichis stored in the Vocabulary subsystem such as shown in FIG. 10 , and maybe accessed in the Context Awareness, Machine Learning and Output andAction systems and methods in FIG. 10 and FIG. 11 .

FIG. 12 is a table 1210 showing example input/output (I/O) andassociated information for a nonverbal multi-input and feedback devicesuch as herein. It may be blocks or processing for a portion of thedevice such as a processing portion of the device. The I/O examplesinclude tap, swipe, breath, head, face, eye, peripheral neuron, andcentral neuron tracking. The information includes information for oridentifying a timer, sensor, method, state changes, device, gestureexample, default meaning, software, processing, whether a match is made(e.g., Yes), metadata and enriched action. The information may includesignals used during processing to determine er intended computer controland communication of text, graphics and audio for nonverbal multi-inputand feedback devices. FIG. 12 may be a table containing a plurality ofsensory devices that can either operate the entire system within itselfor interoperate together on a common universal sensory communication andcontrol system like a platform. The table illustrates that each sensorydevice has very unique sensor technologies ranging from conductivefibers to capacitive touch to camera vision and infrared light to MEMSchips and biophysiological measurement of bioelectric, magnetic or evenultrasound, each with their own unique state change parameters andtime-based analog data acquisition and processing, digital conversionmethods, and user defined or autonomic gestures to be input andrecognized. The table reconciles all of these variable analog to digitalinput signals into a common universal “meaning making machine” or“meaning inferring machine” that correlates these into a common controlinterface for manipulating data on a variety of devices ranging fromvisual displays, audio speakers and haptic vibration actuators. Theprocess is software running on a device that can process, feature match,enrich data, and direct customizable outputs based on the‘meaning+context’ of the data. This table attempts to simplify and teachhow a singular system architecture, devices and software, could be builtto allow for any kind of sensor to be used to product communication andcontrol signals and output commands individually or together via what'scalled Sensor Fusion. A single platform that can enable variable sensorsto interoperate, and deliver a consistent, reliable user experienceacross devices.

FIG. 13 is a diagram 1300 showing flow diagram and related input outputexample of tap and/or swipe tracking for a nonverbal multi-input andfeedback device such as herein. It may be for tracking of physical touchand/or proximity via one or more sensors of the device. Diagram 1300 mayapply to a user holding or wearing such a device with sensor that is atouch sensitive display (see green touchscreen), such as a smartphone,tablet or smartwatch, interacts with the displayed information viaproximity and/or pressure sensitive touch inputs. The analog streamingdata is acquired by the touch sensors, and digitally processed, eitherdirectly on the sensory device or via a remotely connected subsystem.The system may include embedded software on the sensory device thathandles the pre-processing of the analog signal. The system may includeembedded software that handles the digitization and post-processing ofthe signals. Post-processing may include but not be limited to variousmodels of compression, feature analysis, classification, metadatatagging, categorization. The system may handle preprocessing, digitalconversion, and post-processing using a variety of methods, ranging fromstatistical to machine learning. As the data is digitallypost-processed, system settings and metadata may be referred todetermine how certain logic rules in the application are to operate,which may include mapping certain signal features to certain actions.Based on these mappings, the system operates by executing commands andmay include saving data locally on the sensory device or another storagedevice, streaming data to other subsystems or networks.

In the case illustrated in FIG. 13 , at steps 1310-1360, the user islooking at a display that may include characters, symbols, pictures,colors, videos, live camera footage or other visual, oral or interactivecontent. In one example, at 1310 the device receives as input, inputgesture 1—Double Tap which is processed and the device outputs at 1320,Output 1—“I want” displayed full screen. In another example, at 1330 thedevice receives as input, input gesture 2 a—Swipe—start, then inputgesture 2 b—Swipe—middle, then input gesture 2 c—Swipe—end release.These inputs are processed and the device outputs at 1360, Output2—“Help” displayed full screen.

In this example, the user is looking at a mobile tablet or smartphone.The user has been presented a blank ‘gesture canvas’ with a selection ofconfigurable sequential tap and directional swipe gestures to choosefrom that are each mapped to a user's preferred communication attributesbased on typical functional communication. User can rapidly compose aphrase by executing either a single gesture as a shortcut for an entirephrase, or a sequence of tap, swipe and/or movement gestures to build aphrase. The user progressively chooses the next word until they aresatisfied with the phrase they have composed and can determine how toactuate the phrase. Algorithms may be used to predict the nextcharacter, word, or phrase, and may rearrange or alter the expressiondepending on its intended output including but not limited to appendingemoji, symbols, colors, sounds or rearranging to correct for spelling orgrammar errors. The user may desire for the phrase to be spoken aloud toa person nearby, thus selecting a “play button” or simply allowing thesentence to time out to be executed automatically. If they compose aphrase that is a control command like “turn off the lights”, they canselect a “send button” or may, based on semantic natural languageprocessing and understanding, automatically send the phrase to a thirdparty virtual assistant system to execute the command, and turn off thelights. The potential use of metadata, in this example, could simply begeolocation data sourced from other systems such as GIS or GPS data orWIFI data, or manually personalized geofencing in the applicationpersonalization settings, where the system would know if the user is “athome” or “away from home”. In this case, the metadata may play a role inadapting the language being output to reflect the context of the user.For instance, the system could be configured to speak aloud when at homebut send to a caregiver via text message and append GPS coordinates whenaway from home. The system may support collecting and processinghistorical data from the sensory device, system, subsystems, and outputactions to improve the performance and personalization of the system,subsystems, and sensory devices.

In an alternative flow, sensory device 130 has sensors that detect whena user does similar actions as noted above but that combine axial wristmovements plus tap & swipe patterns as shown by the watch type device at“ALT.”

FIG. 14 is a diagram 1400 showing flow diagram and related input outputexample of breathe tracking for a nonverbal multi-input and feedbackdevice such as herein. It may be for tracking of increasing anddecreasing air pressure via one or more sensors of the device. Diagram1400 may apply to a user using an air flow and air pressure sensorydevice that includes a tube that the user sips or puffs into to producepositive and negative air pressure and air flow that is detected by asensor and a valve. The device can be mechanical or digital, andmeasures the amount of air flow and pressure in realtime. The userproduces breath patterns that the sensory device interprets, matches andsends a digital signal to a universal sensory communication system toexecute a command. The analog streaming data is acquired by the air flowand air pressure sensory device, and digitally processed, eitherdirectly on the sensory device or via a remotely connected subsystem.The system may include embedded software on the sensory device thathandles the pre-processing of the analog signal. The system may includeembedded software that handles the digitization and post-processing ofthe signals. Post-processing may include but not be limited to variousmodels of compression, feature analysis, classification, metadatatagging, categorization. The system may handle preprocessing, digitalconversion, and post-processing using a variety of methods, ranging fromstatistical to machine learning. As the data is digitallypost-processed, system settings and metadata may be referred todetermine how certain logic rules in the application are to operate,which may include mapping certain signal features to certain actions.Based on these mappings, the system operates by executing commands andmay include saving data locally on the sensory device or another storagedevice, streaming data to other subsystems or networks.

In the case illustrated in FIG. 14 , at steps 1410-1460, the user islooking at a display (see green touchscreen) that may includecharacters, symbols, pictures, colors, videos, live camera footage orother visual, oral or interactive content. In one example, at 1410-1450the device receives as input, the following sequence: Sensory Device—InUse; Sensory Device—Air Flow & Pressure Sensor; Input gesture 1—Scanning“puff”; Input gesture 2—Scanning “puff”; Input gesture 3—Select—“puff”.These inputs are processed and the device outputs at 1460, Output1—“Help” Displayed full screen.

In this example, the user is looking at a smartphone display. The userhas been presented a set of words to choose from based on typicalfunctional communication with suggested fringe words and access topredictive keyboard and can rapidly compose a phrase by selecting thenext desired word presented in the sentence building interface, oradding a new word manually. The sensory device is sending a basic switchcommand to have the application scan horizontally, puff to stopscanning, and start scanning vertically, puff to stop scanning, creatingan X/Y coordinate intersection over a word. The user then puffs again toselect the word. The user progressively chooses the next word untilthey're satisfied with the phrase they've composed and can determine howto actuate the phrase. Algorithms may be used to predict the nextcharacter, word, or phrase, and may rearrange or alter the expressiondepending on it's intended output including but not limited to appendingemoji, symbols, colors, sounds or rearranging to correct for spelling orgrammar errors. The user may desire for the phrase to be spoken aloud toa person nearby, thus selecting a “play button” or simply allowing thesentence to time out to be executed automatically. If they compose aphrase that is a control command like “turn off the lights”, they canselect a “send button” or may, based on semantic natural languageprocessing and understanding, automatically send the phrase to a thirdparty virtual assistant system to execute the command, and turn off thelights. The potential use of metadata, in this example, could simply begeolocation data sourced from other systems such as GIS or GPS data orWIFI data, or manually personalized geofencing in the applicationpersonalization settings, where the system would know if the user is “athome” or “away from home”. In this case, the metadata may play a role inadapting the language being output to reflect the context of the user.For instance, the system could be configured to speak aloud when at homebut send to a caregiver via text message and append GPS coordinates whenaway from home. The system may support collecting and processinghistorical data from the sensory device, system, subsystems, and outputactions to improve the performance and personalization of the system,subsystems, and sensory devices.

FIG. 15 is a diagram 1500 showing flow diagram and related input outputexample of face and eye tracking for a nonverbal multi-input andfeedback device such as herein. It may be for tracking of facial featuremapping and shape transforms via one or more sensors of the device.Diagram 1500 may apply to a user looking at a visual display—either asmartphone, mobile tablet, augmented reality, virtual reality, mixedreality, computer monitor or television. The sensory device may includea camera and an infrared light projector. This hardware is preferred tobe integrated into the display device, but may be connected as aseparate accessory. The infrared light is projected onto the user'sbody, face or more narrowly limited to specific facial areas like theeyes, nose, brows, mouth or cheeks. The infrared light may have aplurality of beams projected onto the face in order to produce atopographical mesh of which points may be mapped and analyzed by thecamera and system. The speed of which this analysis occurs may becontinuous, at high speed or low speed, or intermittent. The analogstreaming data is acquired by the camera, and digitally processed,either directly on the sensory device or via a remotely connectedsubsystem. The system may include embedded software on the sensorydevice that handles the pre-processing of the analog signal. The systemmay include embedded software that handles the digitization andpost-processing of the signals. Post-processing may include but not belimited to various models of compression, feature analysis,classification, metadata tagging, categorization. The system may handlepreprocessing, digital conversion, and post-processing using a varietyof methods, ranging from statistical to machine learning. As the data isdigitally post-processed, system settings and metadata may be referredto determine how certain logic rules in the application are to operate,which may include mapping certain signal features to certain actions.Based on these mappings, the system operates by executing commands andmay include saving data locally on the sensory device or another storagedevice, streaming data to other subsystems or networks.

In the case illustrated in FIG. 15 , at steps 1510-1550, the user islooking at a display (see green touchscreen) that may includecharacters, symbols, pictures, colors, videos, live camera footage orother visual, oral or interactive content. In one example, at 1510-1540the device receives as input, the following sequence: Sensory Device—useshowing camera calibration of facial features; Sensory Device—Camera &Infrared light projection and software for mapping facial features;Universal Sensory Communication System conducts complex calculations ofrealtime facial transformation data to determine cursor position,intended gesture direction, and end-of-gesture confirmation functions;Input gesture 1—Glance at item, Input gesture 2—Blink eyes. These inputsare processed and the device outputs at 1550, Output 1—“Help” Displayedfull screen.

In this example, the user is looking at a mobile tablet that includesinfrared light projection and a camera to map the user's face inrealtime, calibrating movements and transformations of facial featuresand environmental conditions like ambient light changes. The user hasbeen presented a set of words to choose from based on typical functionalcommunication with suggested fringe words and access to predictivekeyboard and saved phrases and can rapidly compose a phrase by selectingthe next desired word presented in the sentence builder, or adding a newword manually by selecting and using a predictive keyboard. In order toselect an item, the user can move the direction of their face towards anitem grossly moving a cursor towards the intended object, then may useeye movement to fine tune their cursor control. In this example, theuser may blink to select an item or hold their gaze on an item for a setduration of time to select it. The user progressively chooses the nextword until they're satisfied with the phrase they've composed and candetermine how to actuate the phrase. Algorithms may be used to predictthe next character, word, or phrase, and may rearrange or alter theexpression depending on it's intended output including but not limitedto appending emoji, symbols, colors, sounds or rearranging to correctfor spelling or grammar errors. The user may desire for the phrase to bespoken aloud to a person nearby, thus selecting a “play button” orsimply allowing the sentence to time out to be executed automatically.If they compose a phrase that is a control command like “turn off thelights”, they can select a “send button” or may, based on semanticnatural language processing and understanding, automatically send thephrase to a third party virtual assistant system to execute the command,and turn off the lights. The potential use of metadata, in this example,could simply be geolocation data sourced from other systems such as GISor GPS data or WIFI data, or manually personalized geofencing in theapplication personalization settings, where the system would know if theuser is “at home” or “away from home”. In this case, the metadata mayplay a role in adapting the language being output to reflect the contextof the user. For instance, the system could be configured to speak aloudwhen at home but send to a caregiver via text message and append GPScoordinates when away from home. The system may support collecting andprocessing historical data from the sensory device, system, subsystems,and output actions to improve the performance and personalization of thesystem, subsystems, and sensory devices.

FIG. 16 is a diagram 1600 showing flow diagram and related input outputexample of head and neck tracking for a nonverbal multi-input andfeedback device such as herein. It may be for comparing 3D axialposition (X/Y/Z) to a baseline position (0/0/0) via one or more sensorsof the device. Diagram 1600 may apply to a user wearing an augmentedreality headset that includes a display, speakers and vibration hapticmotors and an accelerometer/gyroscope and magnetometer. The user maycalibrate the headset based on the most comfortable and stable neck andhead position which establishes the X/Y/Z position of 0/0/0. Based onthis central ideal position, the user interface is adjusted to conformto the user's range of motion limitations, with an emphasis of reducingthe amount of effort and distance required to move a virtual pointer inaugmented reality from the 0/0/0 position to outer limits of their fieldof view and range of motion The System may be personalized with variousergonomic settings to offset and enhance the user's ease of use andcomfort using the system. The analog streaming data is acquired by themotion sensors in real-time, and digitally processed, either directly onthe sensory device or via a remotely connected subsystem. The system mayinclude embedded software on the sensory device that handles thepre-processing of the analog signal. The system may include embeddedsoftware that handles the digitization and post-processing of thesignals. Post-processing may include but not be limited to variousmodels of compression, feature analysis, classification, metadatatagging, categorization. The system may handle preprocessing, digitalconversion, and post-processing using a variety of methods, ranging fromstatistical to machine learning. As the data is digitallypost-processed, system settings and metadata may be referred todetermine how certain logic rules in the application are to operate,which may include mapping certain signal features to certain actions.Based on these mappings, the system operates by executing commands andmay include saving data locally on the sensory device or another storagedevice, streaming data to other subsystems or networks.

In the case illustrated in FIG. 16 , at steps 1610-1660, the user islooking at a display that may include characters, symbols, pictures,colors, videos, live camera footage or other visual, oral or interactivecontent. In one example, at 1610-1640 the device receives as input, thefollowing sequence: Biomechanics—axial neck & head motion; SensoryDevice—Headmounted Augmented Reality Display with anaccelerometer/gyroscope; Input 1—rotate head in various directions tobuild sentence; Output 1 a—if at home, “I want help” is spoken &displayed full screen on the AR headset. These inputs are processed andat 1650-1660 the device outputs the following sequence: Output 1b—Context Aware (away from home)—System redirects message to cloud,merging geolocation data; and Output 1 b—message modified to “Help, I'mat Oak Park” and sent to friend.

In this example, the user is looking at a set of “radial menus” orcollection of boxes or circles with data in each one that may be asymbol, character, letter, word or entire phrase. The user has beenpresented a set of words that surround a central phrase starter word inthe middle like a hub and spoke to choose from based on typicalfunctional communication with suggested fringe words and access topredictive keyboard, structured and unstructured language. The user canrapidly compose a phrase by selecting the next desired word presented inthe radial menus, or adding a new word manually via another inputmethod. The user traverses the interface using head movement gestures,similar to 3-dimensional swipe movements, to compose communication. Theuser progressively chooses the next word until they're satisfied withthe phrase they've composed and can determine how to actuate the phrase.Algorithms may be used to predict the next character, word, or phrase,and may rearrange or alter the expression depending on its intendedoutput including but not limited to appending emoji, symbols, colors,sounds or rearranging to correct for spelling or grammar errors. Theuser may desire for the phrase to be spoken aloud to a person nearby,thus selecting a “play button” or simply allowing the sentence to timeout to be executed automatically. If they compose a phrase that is acontrol command like “turn off the lights”, they can select a “sendbutton” or may, based on semantic natural language processing andunderstanding, automatically send the phrase to a third party virtualassistant system to execute the command, and turn off the lights. Thepotential use of metadata, in this example, could simply be geolocationdata sourced from other systems such as GIS or GPS data or WIFI data, ormanually personalized geofencing in the application personalizationsettings, where the system would know if the user is “at home” or “awayfrom home”. In this case, the metadata may play a role in adapting thelanguage being output to reflect the context of the user. For instance,the system could be configured to speak aloud when at home but send to acaregiver via text message and append GPS coordinates when away fromhome. The system may support collecting and processing historical datafrom the sensory device, system, subsystems, and output actions toimprove the performance and personalization of the system, subsystems,and sensory devices.

FIG. 17 is a diagram 1700 showing flow diagram and related input outputexample of brain and biosignal tracking for a nonverbal multi-input andfeedback device such as herein. It may be for realtime monitoring ofbiosignal activity, detecting specific time-domain data events via oneor more biosignal sensors of the device. Diagram 1700 may apply to auser wearing an EEG-based brain-computer interface headset containingelectrodes that are contacting the scalp. The electrodes are connectedto an amplifier and analog-to-digital processing pipeline. The sensorydevice (BCI) acquires streaming electrical current data measured inmicrovolts (mV). The more electrodes connected to the scalp and to theBCI, the more streaming analog data can be acquired from the brainwaveactivity. The analog streaming data is acquired by the electrodes,pre-processed through amplification, and digitally processed, eitherdirectly on the sensory device or via a remotely connected subsystem.The system may include embedded software on the sensory device thathandles the pre-processing of the analog signal. The system may includeembedded software that handles the digitization and post-processing ofthe signals. Post-processing may include but not be limited to variousmodels of compression, feature analysis, classification, metadatatagging, categorization. The system may handle preprocessing, digitalconversion, and post-processing using a variety of methods, ranging fromstatistical to machine learning. As the data is digitallypost-processed, system settings and metadata may be referred todetermine how certain logic rules in the application are to operate,which may include mapping certain signal features to certain actions.Based on these mappings, the system operates by executing commands andmay include saving data locally on the sensory device or another storagedevice, streaming data to other subsystems or networks.

In the case illustrated in FIG. 17 , at steps 1710-1750, the user islooking at a display that may include characters, symbols, pictures,colors, videos, live camera footage or other visual, oral or interactivecontent. In one example, at 1710-1740 the device receives as input, thefollowing sequence: Sensory Device—In Use (looking at visual displaywith visual evoked potential stimulation frequencies); SensoryDevice—EEG electrodes and BCI (brain computer interface) on visualcortex (Occipital); Data processing—each interface item is modulating atvarious unique time-domain frequencies (see different color bars of step1730), BCI knows which item has which frequency; Input 1—Each item has aunique frequency attribute, and can be selected by visually fixatingupon it within the user's peripheral and foveal field of view. User canassemble a sentence, and back up or traverse a corpus of language, orcontrol a predictive keyboard. These inputs are processed and the deviceoutputs at 1750, Output 1—“I want Help” can be spoken aloud, sent toanother system over a network, or saved for future re-use.

In this example, the user is looking at a group of concentric circles,arranged in a radial layout, with characters on each circle. The userhas been presented a set of words to choose from based on typicalfunctional communication with suggested fringe words and access topredictive keyboard and can rapidly compose a phrase by selecting thenext desired word presented in the outer ring of circles, or adding anew word manually. The user progressively chooses the next word untilthey're satisfied with the phrase they've composed and can determine howto actuate the phrase. Algorithms may be used to predict the nextcharacter, word, or phrase, and may rearrange or alter the expressiondepending on it's intended output including but not limited to appendingemoji, symbols, colors, sounds or rearranging to correct for spelling orgrammar errors. The user may desire for the phrase to be spoken aloud toa person nearby, thus selecting a “play button” or simply allowing thesentence to time out to be executed automatically. If they compose aphrase that is a control command like “turn off the lights”, they canselect a “send button” or may, based on semantic natural languageprocessing and understanding, automatically send the phrase to a thirdparty virtual assistant system to execute the command, and turn off thelights. The potential use of metadata, in this example, could simply begeolocation data sourced from other systems such as GIS or GPS data orWIFI data, or manually personalized geofencing in the applicationpersonalization settings, where the system would know if the user is “athome” or “away from home”. In this case, the metadata may play a role inadapting the language being output to reflect the context of the user.For instance, the system could be configured to speak aloud when at homebut send to a caregiver via text message and append GPS coordinates whenaway from home. The system may support collecting and processinghistorical data from the sensory device, system, subsystems, and outputactions to improve the performance and personalization of the system,subsystems, and sensory devices.

FIG. 18 is a diagram 1800 showing flow diagram and related input outputexample of multimodal combination of sensors (e.g., tracking and/orprocessing) for a nonverbal multi-input and feedback device such asherein. It may be for using multiple access methods together for inputdata received via one or more sensors of the device. Diagram 1800 mayapply to a user wearing an augmented reality headset combined with abrain computer interface on their head. The headset contains numeroussensors as a combined sensory device including motion and orientationsensors and temporal bioelectric data generated from the brain detectedvia EEG electrodes contacting the scalp of the user, specifically in theregions where visual, auditory and sensory/touch is processed in thebrain. The AR headset may produce visual, auditory and/or hapticstimulation that is detectible via the brain computer interface, and byprocessing brainwave data with motion data, the system may provide newkinds of multi-modal capabilities for a user to control the system. Theanalog streaming data is acquired by the Accelerometer, Gyroscope,Magnetometer and EEG analog-to-digital processor, and digitallyprocessed, either directly on the sensory device or via a remotelyconnected subsystem. The system may include embedded software on thesensory device that handles the pre-processing of the analog signal. Thesystem may include embedded software that handles the digitization andpost-processing of the signals. Post-processing may include but not belimited to various models of compression, feature analysis,classification, metadata tagging, categorization. The system may handlepreprocessing, digital conversion, and post-processing using a varietyof methods, ranging from statistical to machine learning. As the data isdigitally post-processed, system settings and metadata may be referredto determine how certain logic rules in the application are to operate,which may include mapping certain signal features to certain actions.Based on these mappings, the system operates by executing commands andmay include saving data locally on the sensory device or another storagedevice, streaming data to other subsystems or networks.

In the case illustrated in FIG. 18 , at steps 1810-1850, the user islooking at a display that may include characters, symbols, pictures,colors, videos, live camera footage or other visual, oral or interactivecontent. In one example, at 1810-1840 the device receives as input, thefollowing sequence: Sensory Device—In Use (looking at visual displaywith visual evoked potential stimulation frequencies) combined with EEGbased BCI; Sensory Device—EEG electrodes and BCI (brain computerinterface) on visual cortex (Occipital), plus processing neck & headmovement data; Sensory Device—EEG electrodes and BCI (brain computerinterface) on visual cortex (Occipital), plus processing neck & headmovement data; Data processing—each interface item is modulating atvarious unique time-domain frequencies, BCI knows which item has whichfrequency (see different color bars of step 1820). User may use eithermovement or mental fixation, or both combined to make selections in theinterface; Input 1 a—User makes slight neck & head movement with limitedrange of motion to select “I want”; Input 1 b—User fixates on the outerdistant item to select it, completing the phrase, “I want help”. Theseinputs are processed and the device outputs at 1850, Output 1—Systemsays “I want help” and displays upon the device's AR HMD Display.

In this example, the user is looking at a visual menu system in AR withcertain hard to reach elements flickering at different frequencies. Theuser has been presented a set of items to choose from based on typicalfunctional communication with suggested fringe words and access topredictive keyboard and can rapidly compose a phrase by selecting thenext desired word presented in the AR head mounted display, or adding anew word manually. Enabling the user affordances of extra-sensory reachof visible objects out of reach within the comfortable range of motionof neck movement. The user progressively chooses the next word untilthey're satisfied with the phrase they've composed and can determine howto actuate the phrase. Algorithms may be used to predict the nextcharacter, word, or phrase, and may rearrange or alter the expressiondepending on its intended output including but not limited to appendingemoji, symbols, colors, sounds or rearranging to correct for spelling orgrammar errors. The user may desire for the phrase to be spoken aloud toa person nearby, thus selecting a “play button” or simply allowing thesentence to time out to be executed automatically. If they compose aphrase that is a control command like “turn off the lights”, they canselect a “send button” or may, based on semantic natural languageprocessing and understanding, automatically send the phrase to a thirdparty virtual assistant system to execute the command, and turn off thelights. The potential use of metadata, in this example, could simply begeolocation data sourced from other systems such as GIS or GPS data orWIFI data, or manually personalized geofencing in the applicationpersonalization settings, where the system would know if the user is “athome” or “away from home”. In this case, the metadata may play a role inadapting the language being output to reflect the context of the user.For instance, the system could be configured to speak aloud when at homebut send to a caregiver via text message and append GPS coordinates whenaway from home. The system may support collecting and processinghistorical data from the sensory device, system, subsystems, and outputactions to improve the performance and personalization of the system,subsystems, and sensory devices.

FIG. 19 is a flow diagram 1900 showing a closed loop biosignal data flowfor a nonverbal multi-input and feedback device such as herein. It maybe performed by inputs or a computer of the device. The flow 1900includes computer stimulates visual, auditory and somatosensory cortexwith evoked potentials; signal processing of realtime streaming brainresponse; human controls computer based on mental fixation ofstimulation frequencies; and system can determine different output oractions on behalf of the user for input data received via one or moresensors of the device. Diagram 1900 may apply to a user wearing any ofthe nonverbal multi-input and feedback devices and/or sensors herein. Asa result of this being closed-loop biofeedback and sensory communicationand control system that stimulates the brains senses of sight, sound,and touch and reads specific stimulation time-based frequencies, andtags them with metadata in real-time as the analog data is digitized,the user can rapidly learn how to navigate and interact with the systemusing their brain directly. This method of reinforcement learning isknown in the rapid development process of the brain's patternrecognition abilities and the creation of neural plasticity to developnew neural connections based on stimulation and entrainment. Thisfurther enables the system to become a dynamic neural prostheticextension of their physical and cognitive abilities. The merging ofcontext-awareness metadata, vocabulary, and output and action logic intothe central application in addition to a universal interface for signalacquisition and data processing is what makes this system extremelyspecial. Essentially, this system helps reduce the time latency betweendetecting cognitive intention and achieving the associated desiredoutcome, whether that be pushing a button, saying a word or controllingrobots, prosthetics, smart home devices or other digital systems.

FIG. 20 is a flow diagram 2000 showing multimodal, multi-sensory systemfor communication and control for a nonverbal multi-input and feedbackdevice such as herein. It may be performed by inputs or a computer ofthe device. The flow 2000 includes synchronizing signals from multiplebiosensors including brain, body (see skin colored arm), eye andmovement; processing multiple models concurrently for multi-sensoryinput; and directing and processing biofeedback through peripheralsubsystems. Diagram 2000 may apply to a user wearing any of thenonverbal multi-input and feedback devices and/or sensors herein.

FIG. 21 is a block diagram 2100 showing an example of cloud processingfor a nonverbal multi-input and feedback device such as herein. It hasthe cloud system, the nonverbal multi-input device and an authorizationsystem. Diagram 2100 includes: machine Learning processing signal dataon device; metadata enrichment; push raw and processed data to cloud;cloud application building new models for devices; system updatesdevices remotely and wirelessly; secure and privacy compliant. Thisconfiguration is quite powerful but unassumingly simple in this blockdiagram.

FIG. 22 is a block diagram 2200 showing an example of a systemarchitecture for integrated virtual AI assistant and web services for anonverbal multi-input and feedback device such as herein. Diagram 2200includes: system manages intention signal acquisition, processing,language composition, and output; in the event where a user wants tosend their intention to a virtual assistant (like Alexa, Siri). Theblocks outside of the dashed border run on the sensory device, andcurrently, the blocks inside the dashed line are running in the cloud(e.g., represent a custom configuration for how to using the Alexaservice in a cloud architecture.) It could also be possible that all ofwhat's described here as in the cloud could run locally in the sensorydevice

FIG. 23 is a block diagram 2300 showing an example of system operationsfor a nonverbal multi-input and feedback device such as herein. Diagram2300 includes: system operation blocks including authentication. This isan example of the complexity of a system operating in the cloud.Everything in this figure is in the cloud, except for the applicationthat is running on the sensory device.

Closing Comments

Throughout this description, the embodiments and examples shown shouldbe considered as exemplars, rather than limitations on the apparatus andprocedures disclosed or claimed. Although many of the examples presentedherein involve specific combinations of method acts or system elements,it should be understood that those acts and those elements may becombined in other ways to accomplish the same objectives. With regard toflowcharts, additional and fewer steps may be taken, and the steps asshown may be combined or further refined to achieve the methodsdescribed herein. Acts, elements and features discussed only inconnection with one embodiment are not intended to be excluded from asimilar role in other embodiments.

As used herein, “plurality” means two or more. As used herein, a “set”of items may include one or more of such items. As used herein, whetherin the written description or the claims, the terms “comprising”,“including”, “carrying”, “having”, “containing”, “involving”, and thelike are to be understood to be open-ended, i.e., to mean including butnot limited to. Only the transitional phrases “consisting of” and“consisting essentially of”, respectively, are closed or semi-closedtransitional phrases with respect to claims. Use of ordinal terms suchas “first”, “second”, “third”, etc., in the claims to modify a claimelement does not by itself connote any priority, precedence, or order ofone claim element over another or the temporal order in which acts of amethod are performed, but are used merely as labels to distinguish oneclaim element having a certain name from another element having a samename (but for use of the ordinal term) to distinguish the claimelements. As used herein, “and/or” means that the listed items arealternatives, but the alternatives also include any combination of thelisted items.

It is claimed:
 1. A method for associating communication commands withmultiple new nonverbal user's gestures, the method comprising: receivingthe multiple nonverbal user's gestures detected by at least one sensoron a sensory device, wherein the user's gestures include at least one ofbrainwaves having a time based analog signal, head and neck movementhaving a time based movement, or eyeglances having a time basedmovement, wherein the user's gestures are new user's gestures that arenot pre-configured gestures and that do not exist in a database of thesensory device; saving a representation of the user's gestures in thedatabase; associating the representation of the user's gestures with acommunications command, wherein associating includes: receiving thecommunications command from the user, creating an association betweenthe user's gesture and the communications command; configuring a newpre-configured gesture including the representation of the user'sgestures, the association of the representation of the user's gestureswith the communications command, and the communications command; storingthe new pre-configured gesture in the database; wherein thecommunication command identifies a graphical image that therepresentation of the user's gestures are translated to and a speechcommand comprising spoken natural language of a selected language thatcorresponds to the representation of the user's gestures; the speechcommand comprises a word spoken natural language for the graphical imagethat is translated into the selected language; displaying the graphicalimage and generating an audio signal to output the speech command in theselected language on the sensory device; and transmitting the translatedgraphical image and the translated speech command to another devicedifferent than the sensory device.
 2. The method of claim 1, wherein theuser's gestures include at least two of the brainwaves, the head andneck movement, the eyeglances, taps, swipes, diagonals, motion gestures,and breath gestures.
 3. The method of claim 1, further comprising:receiving a request from the user to create the new pre-configuredgesture prior to receiving the multiple nonverbal user's gestures; andreceiving a request from the user to display a preview of therepresentation of the user's gestures and in response, displaying therepresentation of the user's gestures prior to associating therepresentation of the user's gestures.
 4. The method of claim 1, furthercomprising: analyzing and categorizing the representation of the user'sgestures based on a prior predetermined library of representations ofthe user's gestures and corresponding user communication commands, andstoring a category for the new pre-configured gesture based on acategory determined by the categorizing.
 5. The method of claim 1,wherein the communication command includes contextual data including atleast one of contact list metadata, location metadata, time metadata andurgency metadata.
 6. The method of claim 1, wherein the database is aunique gesture database associated with an account of the user, andwherein the communication command further identifies an actuationpattern that the user corresponds to representation of the user'sgestures.
 7. The method of claim 1, wherein the brainwaves have a timebased analog signal with a beginning middle and end, the head and neckmovement has a time based movement with a beginning, middle and end, andthe eyeglances have a time based movement with a beginning middle andend.
 8. The method of claim 1, wherein transmitting the graphical imageto another device includes transmitting the identified graphical imageand the translated speech command to a communication device located witha second user; and wherein the identified graphical image causes anaction on behalf of the user to be performed on the communicationdevice.
 9. The method of claim 1, wherein associating the representationof the user's gestures includes analyzing and categorizing therepresentation of the user's gestures detected using a cloud system andusing machine learning based on the prior predetermined library; andfurther comprising: prior to receiving the multiple user's gestures,receiving a user selection of the selected language of various languagesin which audio signals of speech commands will be output; and usingmachine learning or AI to derive the representation of the user'sgestures from the user's gestures.
 10. The method of claim 1, whereinthe user's gestures include a tap causing the multiple user's gesturesto be received by the sensor; and wherein the tap is not translated tothe graphical image.
 11. A method for constructing user's actions frommultiple biosignal user's gestures, the method comprising: detecting themultiple biosignal user's gestures by at least one sensor on a sensorydevice, wherein the user's gestures include at least two of brainwaveshaving a time based analog signal, head and neck movement having a timebased movement, and eyeglances having a time based movement; determininga vocabulary model based on at least one language selected by the userin the detected user's gestures; representing a user's context based onthe detected user's gestures, wherein the user's context includesmetadata that is processed from sensor inputs of at least one of abiometric sensor; an environment sensor; an object recognition sensor; afacial recognition sensor; a voice recognition sensor; a date and timesensor; a history sensor; a location sensor; or a proximity sensor;constructing a set of one or more potential actions based on the user'scontext, the vocabulary model and the detected user's gestures, whereinthe set of one or more potential actions includes at least one of text,graphics, emoji, or audio sent in control commands or communication toanother device; presenting the potential actions to the user using thevocabulary model, wherein presenting includes one of displaying thetext, displaying the graphics, displaying the emoji, or playing theaudio; receiving a user selection of a user selected action of thepotential actions for communicating; and communicating the user selectedaction over a network to another device different than the sensordevice.
 12. The method of claim 11, wherein the user's gestures onlyinclude one of the head and neck movement, or the eyeglances.
 13. Themethod of claim 11, wherein the sensory device is an augmented reality(AR) headset, and wherein the user's gestures also include one of facialtracking or microphone audio.
 14. The method of claim 13, wherein theuser's gestures include at least one of EEG (Electroencephalogram)signals or EMG (Electromyogram) signals including processed time-basedfrequency neuronal data that is associated with objects to interact withthem.
 15. The method of claim 11, wherein the user's gestures alsoinclude at least one of facial tracking; eye and pupil tracking,touches, taps, swipes, diagonals, motion gestures or breath gestures.16. The method of claim 11, wherein detecting the user's gesturesincludes: stimulating at least one of the user's visual, auditory orsomatosensory cortex with evoked potentials; signal processing ofrealtime streaming brain response from the user, wherein the brainresponse is the user's gestures; the user controlling the device basedon a mental fixation of the stimulation frequencies, wherein thefixation is part of the brain response; and determining the potentialactions based on the user controlling the device.
 17. The method ofclaim 11, wherein detecting the user's gestures includes using aclosed-loop biofeedback and sensory communication and control systemthat stimulates the brains senses of sight, sound, and touch and readsspecific stimulation time-based frequencies, and tags them with metadatain real-time as the analog data is digitized so that the user canrapidly learn how to navigate and interact with the system directlyusing the user's brain to input the gestures.
 18. The method of claim11, wherein the sensor device is one of: (1) a virtual world (VR)headset, application or computing device, or (2) a real world headset,application or computing device; and wherein the another device is acloud system hosting one of: (1) a virtual reality application, or (2) amixed reality application.
 19. The method of claim 11, whereincommunicating includes transmitting a signal for one of displays,computing devices, controllers, speakers, or network communicationdevices; wherein the sensor device is one of a flat screen display;augmented/virtual reality device; virtual AI assistant; synthesizedvoice device; prosthetic device; social media device or a messagingdevice; and wherein the another device is a host of one of anaugmented/virtual reality application; virtual AI assistant application;synthesized voice application; a prosthetic device application; a socialmedia application or a messaging application.
 20. The method of claim11, wherein constructing includes: categorizing the user's gesturesbased on a prior predetermined library of user's gestures andcorresponding user's actions, and newly received user's gestures andcorresponding user's actions; identifying a graphical image that thepotential actions is translated to and a speech command that correspondsto the potential actions; the speech command comprises at least one wordspoken in the vocabulary model for the graphical image that istranslated into the vocabulary model; wherein presenting includesdisplaying the graphical image and generated audio signal to output thespeech command in the selected language on the sensory device; andwherein communicating includes transmitting the identified graphicalimage and the translated speech command to a communication devicelocated with a second user; and wherein the identified graphical imagecauses an action on behalf of the user to be performed on thecommunication device.
 21. The method of claim 11, wherein the brainwaveshave a time based analog signal with a beginning middle and end, thehead and neck movement has a time based movement with a beginning,middle and end, and the eyeglances have a time based movement with abeginning middle and end; and wherein constructing includes analyzingand categorizing the multiple detected user's gestures using a cloudsystem and using machine learning based on the prior predeterminedlibrary; and further comprising: prior to detecting the user's gestures,receiving a user selection of the at least one language selected by theuser in the detected user's gestures.